Variance and Informativity in Different Measures of Linguistic Acceptability

In this paper, we deal with the issue of variability in different measures of linguistic acceptability. It has been argued that acceptability, when measured with the magnitude estimation method (ME), reveals the underlying gradience of linguistic judgments, while other measures, like Likert-scale judgments (e.g. on a 7-point scale) do not provide the same amount of information about the gradient basis of linguistic judgments. We will question this assumption by claiming that not only do Likert-scale judgments provide the same amount of information about a given empirical hypothesis, but also that the inherent variability of ME judgments makes them more susceptible to the production of spurious variance. We will back up these claims by reporting data from a study which compares three different measures of linguistic acceptability, categorical, Likert-scale, and ME. This study was concerned with a phenomenon which has attracted much attention in the literature on linguistic judgments, namely word order variation in German. For the empirical hypothesis we investigated, all three measures of acceptability contain the same amount of information relevant to hypothesis testing. In addition, we show that ME judgments contain more spurious variance and hence are more vulnerable to a reduction in statistical power. The next section deals with the issues of gradience and variance in different measures of linguistic acceptability. Section 3 reports on the empirical study. The final section discusses the findings and relates them back to the question of informativity and variability of different acceptability measures.